Providing visualization data to a co-located plurality of mobile devices

ABSTRACT

A computer-implemented method according to one embodiment includes identifying a plurality of mobile devices, sending a unique identifier to each of the plurality of mobile devices, receiving results of mobile vision sensing from each of the plurality of mobile devices, where the results of mobile vision sensing include, for each of the plurality of mobile devices, an identification of the unique identifier of other mobile devices of the plurality of mobile devices that are within a field of view of the mobile device, determining a relative location of each of the plurality of mobile devices, utilizing the results of mobile vision sensing, and assigning visualization data to each of the plurality of mobile devices, based on the relative location of each of the plurality of mobile devices.

BACKGROUND

The present invention relates to data display, and more specifically,this invention relates to analyzing a plurality of mobile devices andproviding visualization data to the plurality of mobile devices, basedon the analysis.

Card stunts (e.g., the coordinated raising of cards by a group ofindividuals in order to create an image or text) are a popular andeffective means of expression. However, current implementations of cardstunts are time and resource intensive. For example, many card stuntstake months to plan, and may require rehearsals and costlyinfrastructure.

SUMMARY

A computer-implemented method according to one embodiment includesidentifying a plurality of mobile devices, sending a unique identifierto each of the plurality of mobile devices, receiving results of mobilevision sensing from each of the plurality of mobile devices, where theresults of mobile vision sensing include, for each of the plurality ofmobile devices, an identification of the unique identifier of othermobile devices of the plurality of mobile devices that are within afield of view of the mobile device, determining a relative location ofeach of the plurality of mobile devices, utilizing the results of mobilevision sensing, and assigning visualization data to each of theplurality of mobile devices, based on the relative location of each ofthe plurality of mobile devices.

According to another embodiment, a computer program product forproviding visualization data to a plurality of mobile devices comprisesa computer readable storage medium having program instructions embodiedtherewith, wherein the computer readable storage medium is not atransitory signal per se, and where the program instructions areexecutable by a processor to cause the processor to perform a methodcomprising identifying the plurality of mobile devices, utilizing theprocessor, sending a unique identifier to each of the plurality ofmobile devices, utilizing the processor, receiving, utilizing theprocessor, results of mobile vision sensing from each of the pluralityof mobile devices, where the results of mobile vision sensing include,for each of the plurality of mobile devices, an identification of theunique identifier of other mobile devices of the plurality of mobiledevices that are within a field of view of the mobile device,determining, utilizing the processor, a relative location of each of theplurality of mobile devices, utilizing the results of mobile visionsensing, and assigning, utilizing the processor, the visualization datato each of the plurality of mobile devices, based on the relativelocation of each of the plurality of mobile devices.

A system according to another embodiment includes a processor and logicintegrated with the processor, executable by the processor, orintegrated with and executable by the processor, the logic beingconfigured to identify a plurality of mobile devices, send a uniqueidentifier to each of the plurality of mobile devices, receive resultsof mobile vision sensing from each of the plurality of mobile devices,where the results of mobile vision sensing include, for each of theplurality of mobile devices, an identification of the unique identifierof other mobile devices of the plurality of mobile devices that arewithin a field of view of the mobile device, determine a relativelocation of each of the plurality of mobile devices, utilizing theresults of mobile vision sensing, and assign visualization data to eachof the plurality of mobile devices, based on the relative location ofeach of the plurality of mobile devices.

Other aspects and embodiments of the present invention will becomeapparent from the following detailed description, which, when taken inconjunction with the drawings, illustrate by way of example theprinciples of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a cloud computing node according to an embodiment of thepresent invention.

FIG. 2 depicts a cloud computing environment according to an embodimentof the present invention.

FIG. 3 depicts abstraction model layers according to an embodiment ofthe present invention.

FIG. 4 illustrates a method for providing visualization data to aco-located plurality of mobile devices, in accordance with oneembodiment.

FIG. 5 illustrates an exemplary card stunt as a service (CaaS)architecture, in accordance with one embodiment.

FIG. 6 illustrates a method for implementing a card stunt as a service(CaaS) mobile application of a mobile device, in accordance with oneembodiment.

FIG. 7 illustrates a method for implementing a card stunt as acloud-side (CaaS) service, in accordance with one embodiment.

FIG. 8 illustrates an exemplary card stunt as a service (CaaS)visualization, in accordance with one embodiment.

FIG. 9 illustrates an example of an observer mobile device and a screenof a target mobile device, in accordance with one embodiment.

FIG. 10 illustrates exemplary incorrect measurements caused by cameraorientations, in accordance with one embodiment.

FIG. 11 illustrates an exemplary Cartesian plane to solve constrainedoptimization, in accordance with one embodiment.

FIG. 12 illustrates an exemplary geometry for an alternative costfunction, in accordance with one embodiment.

DETAILED DESCRIPTION

The following description discloses several preferred embodiments ofsystems, methods and computer program products for providingvisualization data to a co-located plurality of mobile devices. Variousembodiments provide a method to determine a relative location of each ofa plurality of mobile devices, based on observed data, and determine anddeploy visualization data to each of the plurality of mobile devices,based on the relative location, such that the visualization data, whendisplayed by the screens of each of the plurality of mobile devices,creates a composite image (e.g., akin to a card stunt) when the screensof the plurality of mobile devices are viewed by an observer locatedremotely from the mobile devices at a distance where the observer isable to see the plurality of screens.

The following description is made for the purpose of illustrating thegeneral principles of the present invention and is not meant to limitthe inventive concepts claimed herein. Further, particular featuresdescribed herein can be used in combination with other describedfeatures in each of the various possible combinations and permutations.

Unless otherwise specifically defined herein, all terms are to be giventheir broadest possible interpretation including meanings implied fromthe specification as well as meanings understood by those skilled in theart and/or as defined in dictionaries, treatises, etc.

It must also be noted that, as used in the specification and theappended claims, the singular forms “a,” “an” and “the” include pluralreferents unless otherwise specified. It will be further understood thatthe terms “includes” and/or “comprising,” when used in thisspecification, specify the presence of stated features, integers, steps,operations, elements, and/or components, but do not preclude thepresence or addition of one or more other features, integers, steps,operations, elements, components, and/or groups thereof.

The following description discloses several preferred embodiments ofsystems, methods and computer program products for providingvisualization data to a co-located plurality of mobile devices.

In one general embodiment, a computer-implemented method includesidentifying a plurality of mobile devices, determining a relativelocation of each of the plurality of mobile devices, and assigningvisualization data to each of the plurality of mobile devices, based onthe relative location of each of the plurality of mobile devices.

In another general embodiment, a computer program product for providingvisualization data to a plurality of mobile devices comprises a computerreadable storage medium having program instructions embodied therewith,wherein the computer readable storage medium is not a transitory signalper se, and where the program instructions are executable by a processorto cause the processor to perform a method comprising identifying theplurality of mobile devices, utilizing the processor, determining arelative location of each of the plurality of mobile devices, utilizingthe processor, and assigning, utilizing the processor, the visualizationdata to each of the plurality of mobile devices, based on the relativelocation of each of the plurality of mobile devices.

In another general embodiment, a system includes a processor and logicintegrated with the processor, executable by the processor, orintegrated with and executable by the processor, the logic beingconfigured to identify a plurality of mobile devices, determine arelative location of each of the plurality of mobile devices, and assignvisualization data to each of the plurality of mobile devices, based onthe relative location of each of the plurality of mobile devices.

It is understood in advance that although this disclosure includes adetailed description on cloud computing, implementation of the teachingsrecited herein are not limited to a cloud computing environment. Rather,embodiments of the present invention are capable of being implemented inconjunction with any other type of computing environment now known orlater developed.

Cloud computing is a model of service delivery for enabling convenient,on-demand network access to a shared pool of configurable computingresources (e.g. networks, network bandwidth, servers, processing,memory, storage, applications, virtual machines, and services) that canbe rapidly provisioned and released with minimal management effort orinteraction with a provider of the service. This cloud model may includeat least five characteristics, at least three service models, and atleast four deployment models.

Characteristics are as follows:

On-demand self-service: a cloud consumer can unilaterally provisioncomputing capabilities, such as server time and network storage, asneeded automatically without requiring human interaction with theservice's provider.

Broad network access: capabilities are available over a network andaccessed through standard mechanisms that promote use by heterogeneousthin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

Resource pooling: the provider's computing resources are pooled to servemultiple consumers using a multi-tenant model, with different physicaland virtual resources dynamically assigned and reassigned according todemand. There is a sense of location independence in that the consumergenerally has no control or knowledge over the exact location of theprovided resources but may be able to specify location at a higher levelof abstraction (e.g., country, state, or datacenter).

Rapid elasticity: capabilities can be rapidly and elasticallyprovisioned, in some cases automatically, to quickly scale out andrapidly released to quickly scale in. To the consumer, the capabilitiesavailable for provisioning often appear to be unlimited and can bepurchased in any quantity at any time.

Measured service: cloud systems automatically control and optimizeresource use by leveraging a metering capability at some level ofabstraction appropriate to the type of service (e.g., storage,processing, bandwidth, and active user accounts). Resource usage can bemonitored, controlled, and reported providing transparency for both theprovider and consumer of the utilized service.

Service Models are as follows:

Software as a Service (SaaS): the capability provided to the consumer isto use the provider's applications running on a cloud infrastructure.The applications are accessible from various client devices through athin client interface such as a web browser (e.g., web-based e-mail).The consumer does not manage or control the underlying cloudinfrastructure including network, servers, operating systems, storage,or even individual application capabilities, with the possible exceptionof limited user-specific application configuration settings.

Platform as a Service (PaaS): the capability provided to the consumer isto deploy onto the cloud infrastructure consumer-created or acquiredapplications created using programming languages and tools supported bythe provider. The consumer does not manage or control the underlyingcloud infrastructure including networks, servers, operating systems, orstorage, but has control over the deployed applications and possiblyapplication hosting environment configurations.

Infrastructure as a Service (IaaS): the capability provided to theconsumer is to provision processing, storage, networks, and otherfundamental computing resources where the consumer is able to deploy andrun arbitrary software, which can include operating systems andapplications. The consumer does not manage or control the underlyingcloud infrastructure but has control over operating systems, storage,deployed applications, and possibly limited control of select networkingcomponents (e.g., host firewalls).

Deployment Models are as follows:

Private cloud: the cloud infrastructure is operated solely for anorganization. It may be managed by the organization or a third party andmay exist on-premises or off-premises.

Community cloud: the cloud infrastructure is shared by severalorganizations and supports a specific community that has shared concerns(e.g., mission, security requirements, policy, and complianceconsiderations). It may be managed by the organizations or a third partyand may exist on-premises or off-premises.

Public cloud: the cloud infrastructure is made available to the generalpublic or a large industry group and is owned by an organization sellingcloud services.

Hybrid cloud: the cloud infrastructure is a composition of two or moreclouds (private, community, or public) that remain unique entities butare bound together by standardized or proprietary technology thatenables data and application portability (e.g., cloud bursting forload-balancing between clouds).

A cloud computing environment is service oriented with a focus onstatelessness, low coupling, modularity, and semantic interoperability.At the heart of cloud computing is an infrastructure comprising anetwork of interconnected nodes.

Referring now to FIG. 1, a schematic of an example of a cloud computingnode is shown. Cloud computing node 10 is only one example of a suitablecloud computing node and is not intended to suggest any limitation as tothe scope of use or functionality of embodiments of the inventiondescribed herein. Regardless, cloud computing node 10 is capable ofbeing implemented and/or performing any of the functionality set forthhereinabove.

In cloud computing node 10 there is a computer system/server 12, whichis operational with numerous other general purpose or special purposecomputing system environments or configurations. Examples of well-knowncomputing systems, environments, and/or configurations that may besuitable for use with computer system/server 12 include, but are notlimited to, personal computer systems, server computer systems, thinclients, thick clients, hand-held or laptop devices, multiprocessorsystems, microprocessor-based systems, set top boxes, programmableconsumer electronics, network PCs, minicomputer systems, mainframecomputer systems, and distributed cloud computing environments thatinclude any of the above systems or devices, and the like.

Computer system/server 12 may be described in the general context ofcomputer system-executable instructions, such as program modules, beingexecuted by a computer system. Generally, program modules may includeroutines, programs, objects, components, logic, data structures, and soon that perform particular tasks or implement particular abstract datatypes. Computer system/server 12 may be practiced in distributed cloudcomputing environments where tasks are performed by remote processingdevices that are linked through a communications network. In adistributed cloud computing environment, program modules may be locatedin both local and remote computer system storage media including memorystorage devices.

As shown in FIG. 1, computer system/server 12 in cloud computing node 10is shown in the form of a general-purpose computing device. Thecomponents of computer system/server 12 may include, but are not limitedto, one or more processors or processing units 16, a system memory 28,and a bus 18 that couples various system components including systemmemory 28 to processor 16.

Bus 18 represents one or more of any of several types of bus structures,including a memory bus or memory controller, a peripheral bus, anaccelerated graphics port, and a processor or local bus using any of avariety of bus architectures. By way of example, and not limitation,such architectures include Industry Standard Architecture (ISA) bus,Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, VideoElectronics Standards Association (VESA) local bus, and PeripheralComponent Interconnects (PCI) bus.

Computer system/server 12 typically includes a variety of computersystem readable media. Such media may be any available media that isaccessible by computer system/server 12, and it includes both volatileand non-volatile media, removable and non-removable media.

System memory 28 can include computer system readable media in the formof volatile memory, such as random access memory (RAM) 30 and/or cachememory 32. Computer system/server 12 may further include otherremovable/non-removable, volatile/non-volatile computer system storagemedia. By way of example only, storage system 34 can be provided forreading from and writing to a non-removable, non-volatile magnetic media(not shown and typically called a “hard drive”). Although not shown, amagnetic disk drive for reading from and writing to a removable,non-volatile magnetic disk (e.g., a “floppy disk”), and an optical diskdrive for reading from or writing to a removable, non-volatile opticaldisk such as a CD-ROM, DVD-ROM or other optical media can be provided.In such instances, each can be connected to bus 18 by one or more datamedia interfaces. As will be further depicted and described below,memory 28 may include at least one program product having a set (e.g.,at least one) of program modules that are configured to carry out thefunctions of embodiments of the invention.

Program/utility 40, having a set (at least one) of program modules 42,may be stored in memory 28 by way of example, and not limitation, aswell as an operating system, one or more application programs, otherprogram modules, and program data. Each of the operating system, one ormore application programs, other program modules, and program data orsome combination thereof, may include an implementation of a networkingenvironment. Program modules 42 generally carry out the functions and/ormethodologies of embodiments of the invention as described herein.

Computer system/server 12 may also communicate with one or more externaldevices 14 such as a keyboard, a pointing device, a display 24, etc.;one or more devices that enable a user to interact with computersystem/server 12; and/or any devices (e.g., network card, modem, etc.)that enable computer system/server 12 to communicate with one or moreother computing devices. Such communication can occur via Input/Output(I/O) interfaces 22. Still yet, computer system/server 12 cancommunicate with one or more networks such as a local area network(LAN), a general wide area network (WAN), and/or a public network (e.g.,the Internet) via network adapter 20. As depicted, network adapter 20communicates with the other components of computer system/server 12 viabus 18. It should be understood that although not shown, other hardwareand/or software components could be used in conjunction with computersystem/server 12. Examples, include, but are not limited to: microcode,device drivers, redundant processing units, external disk drive arrays,RAID systems, tape drives, and data archival storage systems, etc.

Referring now to FIG. 2, illustrative cloud computing environment 50 isdepicted. As shown, cloud computing environment 50 includes one or morecloud computing nodes 10 with which local computing devices used bycloud consumers, such as, for example, personal digital assistant (PDA)or cellular telephone 54A, desktop computer 54B, laptop computer 54C,and/or automobile computer system 54N may communicate. Nodes 10 maycommunicate with one another. They may be grouped (not shown) physicallyor virtually, in one or more networks, such as Private, Community,Public, or Hybrid clouds as described hereinabove, or a combinationthereof. This allows cloud computing environment 50 to offerinfrastructure, platforms and/or software as services for which a cloudconsumer does not need to maintain resources on a local computingdevice. It is understood that the types of computing devices 54A-N shownin FIG. 2 are intended to be illustrative only and that computing nodes10 and cloud computing environment 50 can communicate with any type ofcomputerized device over any type of network and/or network addressableconnection (e.g., using a web browser).

Referring now to FIG. 3, a set of functional abstraction layers providedby cloud computing environment 50 (FIG. 2) is shown. It should beunderstood in advance that the components, layers, and functions shownin FIG. 3 are intended to be illustrative only and embodiments of theinvention are not limited thereto. As depicted, the following layers andcorresponding functions are provided:

Hardware and software layer 60 includes hardware and softwarecomponents. Examples of hardware components include: mainframes 61; RISC(Reduced Instruction Set Computer) architecture based servers 62;servers 63; blade servers 64; storage devices 65; and networks andnetworking components 66. In some embodiments, software componentsinclude network application server software 67 and database software 68.

Virtualization layer 70 provides an abstraction layer from which thefollowing examples of virtual entities may be provided: virtual servers71; virtual storage 72; virtual networks 73, including virtual privatenetworks; virtual applications and operating systems 74; and virtualclients 75.

In one example, management layer 80 may provide the functions describedbelow. Resource provisioning 81 provides dynamic procurement ofcomputing resources and other resources that are utilized to performtasks within the cloud computing environment. Metering and Pricing 82provide cost tracking as resources are utilized within the cloudcomputing environment, and billing or invoicing for consumption of theseresources. In one example, these resources may include applicationsoftware licenses. Security provides identity verification for cloudconsumers and tasks, as well as protection for data and other resources.User portal 83 provides access to the cloud computing environment forconsumers and system administrators. Service level management 84provides cloud computing resource allocation and management such thatrequired service levels are met. Service Level Agreement (SLA) planningand fulfillment 85 provide pre-arrangement for, and procurement of,cloud computing resources for which a future requirement is anticipatedin accordance with an SLA.

Workloads layer 90 provides examples of functionality for which thecloud computing environment may be utilized. Examples of workloads andfunctions which may be provided from this layer include: mapping andnavigation 91; software development and lifecycle management 92; virtualclassroom education delivery 93; data analytics processing 94;transaction processing 95; and card stunt as a service (CaaS) 96.

Now referring to FIG. 4, a flowchart of a method 400 is shown accordingto one embodiment. The method 400 may be performed in accordance withthe present invention in any of the environments depicted in FIGS. 1-3and 5-7, among others, in various embodiments. Of course, more or lessoperations than those specifically described in FIG. 4 may be includedin method 400, as would be understood by one of skill in the art uponreading the present descriptions.

Each of the steps of the method 400 may be performed by any suitablecomponent of the operating environment. For example, in variousembodiments, the method 400 may be partially or entirely performed byone or more servers, computers, or some other device having one or moreprocessors therein. The processor, e.g., processing circuit(s), chip(s),and/or module(s) implemented in hardware and/or software, and preferablyhaving at least one hardware component may be utilized in any device toperform one or more steps of the method 400. Illustrative processorsinclude, but are not limited to, a central processing unit (CPU), anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA), etc., combinations thereof, or any other suitablecomputing device known in the art.

As shown in FIG. 4, method 400 may initiate with operation 402, where aplurality of mobile devices are identified. In one embodiment, theplurality of mobile devices may be located within a group ofindividuals. For example, each of the plurality of devices may be in thepossession of a unique individual located within a single crowd ofpeople, such that the plurality of devices are distributed throughoutthe group. In another embodiment, each of the plurality of mobiledevices may include a mobile computing device such as a cell phone, alaptop computer, a portable music player, etc.

Additionally, as shown in FIG. 4, method 400 may proceed with operation404, where a relative location of each of the plurality of mobiledevices is determined. In one embodiment, determining the relativelocation of each of the plurality of mobile devices may includedetermining, for each of the plurality of mobile devices, a location ofthe mobile device with respect to the other devices of the plurality ofmobile devices. In one embodiment, determining the relative location ofeach of the plurality of mobile devices may include performing relativelocalization on each of the plurality of mobile devices in order todetermine a co-located group of the mobile devices.

Further, in one embodiment, determining the relative location of each ofthe plurality of mobile devices may be performed utilizing mobile visionsensing. In another embodiment, pair-wise visual observations may beperformed at each of the plurality of mobile devices (e.g., utilizing anapplication installed on each of the mobile devices, etc.) in order toimplement a vision-based device identification technique. For example,for each of the plurality of mobile devices, a unique identifier for themobile device may be encoded utilizing a color-transition sequence, andsuch unique identifier may be displayed as a vision code on a display ofthe mobile device.

Further still, in another embodiment, for each of the plurality ofmobile devices, the mobile device may identify other mobile devices ofthe plurality of mobile devices that are within a field of view of themobile device (e.g., utilizing one or more cameras of the device and thevision codes of each of the plurality of mobile devices, etc.). Forexample, for each of the plurality of mobile devices, the mobile devicemay instruct a user of the mobile device to hold the mobile device abovetheir head and/or shoulders.

In another example, after a predetermined time or once the mobile devicedetermines that is being held above a user's head/shoulders, the mobiledevice may visually identify other mobile devices within the pluralityof mobile devices. In yet another example, one or more of tactile andaudio feedback may be provided to users of each of the plurality ofmobile devices (e.g., in order to guide them to hold their phones in adesired orientation, etc.). In this way, distributed observations of theplurality of mobile devices may be obtained.

Also, in one embodiment, for each of the plurality of mobile devices,imprecise observations/orientations of the mobile device may becorrected during mobile vision sensing. For example, an orientation of adevice may be normalized utilizing one or more of an API of the mobiledevice, a magnetometer of the mobile device, a gravity sensor of themobile device, an accelerometer of the mobile device, etc. In anotherembodiment, for each of the plurality of mobile devices, compensationmay be made for a displacement of a camera from a center of the mobiledevice during mobile vision sensing. In yet another embodiment, for eachof the plurality of mobile devices, compensation may be made for aninaccurate measurement of a horizontal heading of the mobile deviceduring mobile vision sensing.

In still another embodiment, for each of the plurality of mobiledevices, an exposure of a camera and/or a brightness of a camera may beadjusted during mobile vision sensing, based on external lightingconditions. In another embodiment, a number of distinct colors includedwithin the vision codes used during mobile vision sensing may beadjusted (e.g., to maximize bits-per-symbol, etc.), based on externallighting conditions. In yet another embodiment, determining the relativelocation of each of the plurality of mobile devices may include sendingresults of mobile vision sensing from the plurality of mobile devices toa cloud computing environment.

In addition, in one embodiment, determining the relative location ofeach of the plurality of mobile devices may be performed utilizingconstrained optimization. For example, constrained nonlinearoptimization may be performed within a cloud computing environment inorder to reconstruct relative locations of each of the plurality ofmobile devices, using distributed observations received from theplurality of mobile devices. In another example, the cloud computingenvironment may formulate a constrained optimization problem on the setof distributed observations. For instance, a linearly constrainednonlinear programming algorithm may be used to formulate and solve theconstrained optimization problem.

Furthermore, in one embodiment, the constrained optimization may beaccelerated. For example, acceleration may be performed by formulating alinearly constrained quadratic polynomial programming problem, andsolving the problem within the cloud computing environment utilizing aquadratic programming algorithm to obtain an approximate solution. Theapproximate solution may then be sent to the linearly constrainednonlinear programming algorithm as initial values. In this way, thenonlinear programming algorithm may converge in a faster manner.

Further still, in one embodiment, the results of the constrainedoptimization may be used to reconstruct relative locations of each ofthe plurality of mobile devices with respect to each other (e.g., withina crowd of people, etc.), which may result in reconstructed locationsfor each of the plurality of mobile devices.

Also, as shown in FIG. 4, method 400 may proceed with operation 406,where visualization data is assigned to each of the plurality of mobiledevices, based on the relative location of each of the plurality ofmobile devices. In one embodiment, the visualization data may representone or more of textual and graphical output.

Additionally, in one embodiment, the textual and/or graphical output maybe pixelated into a matrix of tiles within the cloud computingenvironment. For example, the textual and/or graphical output may beresampled to fit a number of the plurality of mobile devices in bothhorizontal and vertical directions. In another example, reconstructedlocations of each of the plurality of mobile devices may be quantized tosnap to a virtual uniform grid, and unit grid spacing may be determined.

Further, in one embodiment, each tile within the matrix may be assignedto a mobile device at a location matching a respective location withinthe matrix. For example, each resampled image may be assigned to amobile device having a quantized location at a corresponding gridlocation of a virtual uniform grid. In another embodiment, a single tileor a timed sequence of tiles may be sent to each of the plurality ofdevices at each location within the virtual uniform grid. In yet anotherembodiment, instead of or in addition to tiles, individual cues may besent to a display of each of the plurality of mobile devices thatinstruct a user of each mobile device to manually display one or morecards.

Further still, in one embodiment, a global positioning system (GPS)module located within one or more of the plurality of mobile devices maybe used to refine the determination of the relative location of each ofthe plurality of mobile devices when a number of the plurality of mobiledevices exceeds a threshold (e.g., within large crowds, etc.). Forexample, the GPS module may provide estimated locations of each of theplurality of mobile devices, and additional constraints may be createdwhen solving the constrained optimization problem. As a result, theconstrained optimization problem may be solved with both the originalconstraints (e.g., from local pair-wise observations using vision codes,etc.) and the GPS-obtained locations.

In another embodiment, adjustments may be made during the determinationof the relative location of each of the plurality of mobile devices toaccount for a height of a user of each of the plurality of mobiledevices, as well as ground slope. In yet another embodiment, a divideand conquer strategy may be utilized during the determination of therelative location of each of the plurality of mobile devices when anumber of the plurality of mobile devices exceeds a threshold.

Also, in one embodiment, instructions for display of visualizationmaterials (e.g., cue cards or other colored or decorated cards, etc.)may be assigned to each of the plurality of mobile devices instead ofvisualization data, based on the relative location of each of theplurality of mobile devices.

In this way, a visualization of textual or graphical data may beachieved (e.g., similar to a card stunt, etc.) using a plurality ofdevices collectively. For example, a grouping of the screens of each ofthe plurality of mobile devices may create a composite image (e.g., akinto a card stunt) when the screens of the plurality of mobile devices areviewed by an observer located remotely from the mobile devices at adistance where the observer is able to see the plurality of screens.This may enable collective visualization without rehearsals,pre-planning, or stationary infrastructure.

Now referring to FIG. 5, an exemplary card stunt as a service (CaaS)architecture 500 is shown according to one embodiment. As shown, theCaaS architecture 500 includes a mobile application 502 and a cloud-sideservice 504. In one embodiment, the mobile application 502 may beinstalled on a mobile device. For example, each of a plurality of userslocated within a group may have an instance of the mobile application502 installed on their mobile device. In another embodiment, thecloud-side service 504 may include a cloud computing environment incommunication with the mobile application 502.

Additionally, the mobile application 502 is in communication with ascreen 506, a camera 508, and sensors 510 of a mobile device. The mobileapplication 502 also includes a plurality of modules 512-520 forimplementing CaaS on the mobile device side. The cloud-side service 504includes a back end optimizer 530 containing a constrained optimizationmodule 522, and also includes a front end coordinator 524 that includesa local observation collecting module 526 and a tile dispatching module528.

For example, in order to perform collective user localization, mobilevision sensing may be performed by the mobile application 502. In oneembodiment, a vision code transceiving module 512 may determine a uniquevision code for the mobile device and may output the vision code via thescreen 506. In another embodiment, the vision code recognizing module514 may identify other vision codes (e.g., vision codes from othermobile devices, etc.), utilizing the camera 508, and may measure arelative angle between the observing mobile device and the observedmobile device. In yet another embodiment, the phone orientation module516 may determine an orientation of the mobile device, utilizing thesensors 510 of the mobile device.

In one embodiment, for the generation of the vision code, for real-timevideo processing on a mobile device, temporally coded single coloredframes may be implemented and extended it to be more error resilient andreal-time decodeable in mobile devices. For example, the vision code mayinclude one or more of red, green, and blue frames. In one embodiment,green frames may be delimiters separating repeated transmissions, andblue and red frames may denote one and zero, respectively. Of course,however, any color may be assigned for any purpose. In anotherembodiment, the number of colors may be extensible depending on theexternal lighting conditions. Fixed-duration coding may be used for eachbit; a blue or red frame lasting for a predefined time may represent asingle ‘1’ or ‘0’ symbol, respectively.

In another embodiment, a vision code may include one or more of threeparts: 1) delimiter; 2) encoded identifier of a mobile device; 3)delimiter. Each mobile device may obtain its unique binary identifierfrom the front end coordinator 524. For error resiliency, the identifiermay be encoded utilizing Hamming code or other errordetecting/correcting codes such as CRC or repetition codes.

Further, in one embodiment, each device may generate its own vision codeand may observe others' codes at the same time. For every incoming videoframe, each pixel may be ternary quantized based on its hue channel inHSV space—for example, into red if the pixel's hue value belongs to arange [300,360) or [0,60), into green if in [60,180), and into blue ifin [180,300). Ternary quantization may achieve reliable decoding undervarious conditions.

To help a mobile device search for a valid vision code in real time, onepixel may be sampled out of every 2×2 (or larger) block, which mayachieve real-time search operations (e.g., at 10-25 fps, etc.). Theternary quantized video frames may be buffered. When a delimiter pixelarrives, the temporal bit sequence at this pixel location until theprevious delimiter may be examined; a valid code may be confirmed if thesequence length is equal to the code length and the decoder finds noerror bit. The same vision codes may appear at multiple adjacent pixelsat once; those pixels' centroid may be defined as that vision code'son-picture coordinates. Once a valid vision code and its on-picturecoordinates are found, the observer may identify the target device,compute the angle θ_(ij) in Table 1, and send a tuple (D_(i), D_(j),θ_(ij)) to the cloud-side service 504, where D_(i) and D_(j) are its ownand the target's identifiers, respectively.

Further, in one embodiment, the other vision codes identified by therecognizing module 514 and the orientation of the mobile devicedetermined by the phone orientation module 516 may be sent to acalculating module 518 within the mobile application 502 that maydetermine local observations (such as relative angles and orientations,etc.) utilizing such information.

For example, the mobile application 502 may observe horizontal angulardistances to other devices seen from its camera 508. Consider FIG. 9,which provides an example of an observer mobile device D_(i) and ascreen of a target mobile device D_(j). In one embodiment, given acamera of an observer device D_(i) and a screen of a target deviceD_(j), the horizontal angle θ_(ij) may be defined by two lines: oneperpendicular to the image plane of D_(i)'s camera, and the horizontalprojection of the line connecting D_(i)'s camera center and D_(j)'sscreen center. In another embodiment, visual observation may allow for acalculation of θ_(ij), given D_(i)'s field of view FoV_(i), theresolution of the picture seen from D_(i)'s camera (W_(i), H_(i)), andthe on-picture coordinates of D_(j) seen from D_(i)'s camera (w_(ij),h_(ij)). In yet another embodiment, an operating platform of the mobiledevice may provide FoV_(i) as well W_(i) and H_(i) at runtime.

Table 1 illustrates an exemplary computation of θ_(ij). Of course, itshould be noted that the computation shown in Table 1 is set forth forillustrative purposes only, and thus should not be construed as limitingin any manner.

TABLE 1   first compute the focal distance F_(i) in pixels:  $\quad\mspace{121mu}{F_{i} = {\frac{W_{i}}{2}\left( {\tan\frac{{FoV}_{i}}{2}} \right)^{- 1}{\quad{{{Then}\mspace{14mu}\theta_{ij}\mspace{11mu}{is}\mspace{14mu}{given}\mspace{14mu}{by}\text{:}\mspace{11mu}\theta_{ij}} = {{- \arctan}\; 2\left( {w_{ij},F_{i}} \right){\quad{{{where}\mspace{14mu}\arctan\; 2\left( {y,x} \right)\mspace{14mu}{is}{\mspace{11mu}\;}a\mspace{14mu}{two}\text{-}{variable}\mspace{14mu}{extension}{of}\mspace{14mu}\arctan\mspace{14mu}\frac{y}{x}},{{so}{\mspace{11mu}\;}{that}{\mspace{11mu}\;}{its}\mspace{14mu}{range}\mspace{14mu}{is}\mspace{11mu}{defined}\mspace{14mu}{over}\mspace{14mu}\left( {{- \pi},\pi} \right)}}}}}}}}$

In one embodiment, the leading negative sign in Table 1 may be added tokeep θ_(ij)'s sign consistent with the Cartesian coordinate system. Thismay illustrate one situation when D_(i) is at a perfect landscapeorientation and standing upright, i.e., α_(y)=0, α_(z)=0. In anotherembodiment, a device held in a user's hand may have nonzero orientationangles. In response, θ_(ij) may be calibrated by compensating thedevice's orientation.

FIG. 10 illustrates exemplary incorrect measurements of θ_(ij) caused bycamera orientations. In one embodiment, to reorient w_(ij) and evaluatethe true θ_(ij), an X-Y-Z intrinsic rotation may be defined, where theelement rotation angles α_(x), α_(y), and α_(z) may be obtainedutilizing an application program interface (API) built into the mobiledevice, as well as sensors 510 of the mobile device, including amagnetometer of the mobile device and a gravity sensor of the mobiledevice. Each observing device may apply Z and Y rotation matrices R_(z)and R_(y) to the pre-rotation (h_(ij), w_(ij), z_(ij)), where z_(ij) isthe point where a virtual sphere of the radius F_(i) intersects with aline originating from (h_(ij), w_(ij)) orthogonal to the camera's imageplane.

Table 2 illustrates an exemplary computation of post-rotationcoordinates (h′_(ij), w′_(ij), z′_(ij)). Of course, it should be notedthat the computation shown in Table 2 is set forth for illustrativepurposes only, and thus should not be construed as limiting in anymanner.

TABLE 2 $\quad\begin{matrix}\begin{matrix}\left\lbrack h_{ij}^{\prime} \right. & w_{ij}^{\prime} & {\left. z_{ij}^{\prime} \right\rbrack^{T} = {R_{y}{R_{z}\left\lbrack \begin{matrix}h_{ij} & w_{ij} & {\left. z_{ij} \right\rbrack^{T},{where}}\end{matrix} \right.}}}\end{matrix} \\{{R_{y} = \begin{bmatrix}{\cos\mspace{14mu}\alpha_{y}} & 0 & {\sin\mspace{14mu}\alpha_{y}} \\0 & 1 & 0 \\{{- \sin}\mspace{14mu}\alpha_{y}} & 0 & {\cos\mspace{14mu}\alpha_{y}}\end{bmatrix}},{R_{z} = \begin{bmatrix}{\cos\mspace{14mu}\alpha_{z}} & {{- \sin}\mspace{14mu}\alpha_{z}} & 0 \\{\sin\mspace{14mu}\alpha_{z}} & {\cos\mspace{14mu}\alpha_{z}} & 0 \\0 & 0 & 1\end{bmatrix}}}\end{matrix}$

In one embodiment, the re-oriented θ′_(ij) may be computed using thecomputation in Table 1 and w′_(ij). The mobile application 502 may sendθ′_(ij) to the front end coordinator 524. Note that x-rotation matrixR_(x) may not be applied at local devices, because the back endoptimizer may presume a global Cartesian plane whose negative x-axis isdefined by the common heading of the crowd. To complete the x-rotationwith respect to the back end coordinate system, the front endcoordinator 524 may evaluate α_(x) , i.e., the average angle of allobserving devices' α_(x). The final θ_(ij) may be obtained byθ′_(ij)+(α_(x)−α_(x) ).

Further still, in one embodiment, in order to perform collective userlocalization, location reconstruction may be performed by the cloud-sideservice 504 utilizing constrained optimization sensing. In oneembodiment, the local observation collecting module 526 may receivelocal observations from the calculating module 518, which may then besent to the constrained optimization module 522 of the back endoptimizer 530 to determine relative locations of each mobile device.

Also, in one embodiment, these relative locations may be sent from theconstrained optimization module 522 to the tile dispatching module 528,which may send timed tiles to the use the relative locations to a tiledisplaying module 520 of the mobile application 502, which may thendisplay the tiles via the screen 506 of the mobile device.

For example, as soon as the front end coordinator 524 retrieves all theobservations from the mobile application 502, the back end optimizer 530may start localization. The observations may include a set of orderedtuples (D_(i), D_(j), θ_(ij)), meaning that an observer D_(i) hasvisually identified a target D_(j) at an angle of θ_(ij). Note that theobservation may be unidirectional so this tuple does not necessarilyimply the existence of a tuple (D_(j), D_(i), θ_(ij)).

In a first case, every device may be heading toward an identicaldirection (i.e., identical α_(x) for all devices) and at perfectlandscape orientations (i.e., α_(y)=0, α_(z)=0 for all devices). In thefirst case, a tuple (D_(i), D_(j), θ_(ij)) may be represented on aCartesian plane virtually overlaid over the ground. The coordinates ofthe two devices (x_(i), y_(i)) and (x_(j), y_(j)) as well as thedistance between them may be unknown. All N device locations z=(x₁, y₁,x₂, y₂, . . . , x_(N), y_(N)) may be reconstructed by formulating anoptimization problem that minimizes the cumulative observation errorssubject to feasibility constraints derived from geometries among mobiledevices.

Consider FIG. 11, which provides an exemplary Cartesian plane to solveconstrained optimization. To define a region in which D_(j) may exist inrelative terms of D_(i)'s location, a trapezoidal area may be createdthat is enclosed by four lines L₁, L₂, L₃, and L₄. To consider apossible imprecise measurement of θ_(ij), L₁ and L₂ may have slopes thatare apart from θ_(ij) by −Δθ and +Δθ, respectively. In one embodiment,there may be a maximum distance that D_(i)'s camera can identify D_(j)'scode. Also, D_(i) and D_(j) may be distant by at least a one-personwidth. These two distance constraints may define L₃ and L₄ which may beorthogonal to the line D_(i)D_(j) and may be distant from D_(i) byd_(far) and d_(near) respectively. From the equations of L₁, L₂, L₃, andL₄, linear inequalities may be derived defining the feasible region inwhich (x_(j), y_(j)) would exist.

Table 3 illustrates an exemplary feasible region derivation. Of course,it should be noted that the derivation shown in Table 3 is set forth forillustrative purposes only, and thus should not be construed as limitingin any manner.

TABLE 3   L₁ → y_(j) ≥ tan(θ_(ij) − Δθ)(x − x_(i)) + y_(i) L₂ → y_(j) ≤tan(θ_(ij) + Δθ)(x − x_(i)) + y_(i) $\quad\begin{matrix}\left. L_{3}\rightarrow{x_{j} \leq {{\frac{1}{\tan\mspace{14mu}\left( {\theta_{ij} + \frac{\pi}{2}} \right)}\left\{ {y_{j} - \left( {y_{i} + d_{fy}} \right)} \right\}} + \left( {x_{i} + d_{fx}} \right)}} \right. \\{\left. L_{4}\rightarrow{x_{j} \geq {{\frac{1}{\tan\mspace{14mu}\left( {\theta_{ij} + \frac{\pi}{2}} \right)}\left\{ {y_{j} - \left( {y_{i} + d_{ny}} \right)} \right\}} + \left( {x_{i} + d_{nx}} \right)}} \right.\begin{matrix}{{{{where}\mspace{14mu} d_{fx}} = {d_{far}\mspace{11mu}\cos\mspace{11mu}\theta_{ij}}},{d_{fy} = {d_{far}\mspace{11mu}\sin\mspace{11mu}\theta_{ij}}},} \\{{{d_{nx} = {d_{{near}\;}\cos\mspace{11mu}\theta_{ij}}},{d_{ny} = {d_{near}\mspace{11mu}\sin\mspace{11mu}\theta_{ij}}}}\;}\end{matrix}}\end{matrix}$

Note that alternative forms may be used for the derivation that placesx_(j) on the left-hand side and may take a reciprocal of the originalslope. It is to have a zero slope instead of infinity at θ_(ij)=0.

Next, a cost function to be minimized may be defined. Consider N mobiledevices and the unidirectional observations among a part of them. Let 0denote a set of ordered pairs where a pair (i, j) is an element if andonly if an observation from D_(i) to D_(j) exists. The objective is tofind a vector z=(x₁, y₁, x₂, y₂, . . . , x_(N), y_(N)) minimizing theerror between θ_(ij) and the computed slope angle of D_(i)D_(j) sujectto the derivation in Table 3.

Table 4 illustrates the cost function ƒ and its gradient. Of course, itshould be noted that the cost function ƒ shown in Table 4 is set forthfor illustrative purposes only, and thus should not be construed aslimiting in any manner.

TABLE 4   f(x₁, y₁, . . . , x_(N), y_(N))     $\quad\begin{matrix}{= {\sum\limits_{{({i,j})} \in O}\left\{ {{\arctan\; 2\left( {{y_{j} - y_{i}},{x_{j} - x_{i}}} \right)} - \theta_{ij}} \right\}^{2}}} \\{{\bigtriangledown f} = {\overset{N}{\sum\limits_{i = 1}}\left( {{\frac{\partial f}{\partial x_{i}}{\hat{x}}_{i}} + {\frac{\partial f}{\partial y_{i}}{\hat{y}}_{i}}} \right)}}\end{matrix}$

To make ƒ differentiable, the angular difference may be squared insteadof taking its absolute value. Note that arctan 2(y, x) is differentiableby x and y except where √{square root over (X²+y²)}+x=0. Thissingularity is not in the domain because D_(i) and D_(j) do not overlap,observations are unidirectional, and the Cartesian plane is defined toalways place D_(i) to the left side of D_(j).

In addition, at least one device may be fixed on the Cartesian plane tokeep the problem's domain bounded. Without loss of generality, aconstraint may be added to anchor an arbitrary device at the origin:x₁=0, y₁=0.

Further, in one embodiment, additional constraints may be considered inspecial cases where the crowd formation is partly or wholly rectangular.In a rectangular formation, the people in the first row may know theyare in the first row, as no one else is in front of them. The mobileapplication 502 may be extended to provide a simple interface so thatone can self-declare that they are in the first row. In our coordinatesystem as in Table 4, this self-declaration may be interpreted into thefollowing constraint: for a device D_(k) that self-declares being in thefront row, x_(k)≤∀x∈{h_(i)|1≤i≤N}.

In another embodiment, similar constraints may be constructed for thosewho self-declare that they are in the last row, left, or right-mostcolumn, etc.

Now the optimization problem may be translated into a nonlinearprogramming problem to find z=(x₁, y₁, x₂, y₂, . . . , x_(N), y_(N))minimizing fin Table 4 subject to linear inequality constraints in Table3, optionally front, back, or side row constraints, and the linearequality anchor constraint. This family of problems may be effectivelysolved by a number of different algorithms.

Upon the return of the reconstructed locations from the back endoptimizer 530, the front end coordinator 524 may plan how to visualize agiven image over the crowd. First, the image to be drawn may beresampled to fit the number of mobile devices in both horizontal andvertical directions. Next, the reconstructed locations of the mobiledevices may be quantized to snap to virtual uniform grids. To determinethe proper distance of a unit grid spacing, the front end coordinator524 may collect the distances to 4-way nearest neighbors from eachmobile device, i.e., the nearest neighbor in each of four quadrantscentered at the mobile device: (−45°, 45°], (45°, 135°], (135°, 225°],and (225°, 315°]. A unit grid spacing may be set as the median distanceof all the 4-way nearest neighbors from each mobile device. Last, eachpixel of the resampled image may be assigned to the mobile device whosequantized locations are at the corresponding grid.

In one embodiment, a camera of a mobile device may be displaced from acenter of the device. To address this, the observer's Y-axis coordinatesy_(i) may be replaced with y_(i)+δ_(i) in Table 3 and Table 4, whereδ_(i) denotes a displacement constant defined by the mobile device'sform factor. A table may be implemented to look up δ_(i) per mobiledevice model at runtime. Note that the target's coordinates (x_(j),y_(j)) may remain the same in the equations because the observer seesthe center of the target's display.

Additionally, in one embodiment, gyroscope drift and magnetometerinterferences may affect the compass sensing, and may therefore affectthe horizontal heading α_(x). In order to address these discrepancies inlocal measurement of the devices' horizontal headings, a variable ε_(i)may be introduced that represents an observer-specific error on itshorizontal heading. For a given observer, it may be assumed that ε_(i)would be considered as a static offset for a short period of time. Thusε_(i) may be modeled to be an observer-dependent variable, rather thanan observation-dependent variable. The cost function ƒ may be revised tohave an ability to manipulate ε_(i) within a small interval.

Table 5 illustrates a revised cost function ƒ. Of course, it should benoted that the revised cost function ƒ shown in Table 5 is set forth forillustrative purposes only, and thus should not be construed as limitingin any manner.

TABLE 5   f(x₁, y₁, . . . , x_(N), y_(N), ϵ₁, . . . , ϵ_(N))  $\quad\begin{matrix}{{= {\sum\limits_{{({i,j})} \in O}\left\{ {{{arc}\;\tan\; 2\left\{ {{y_{j} - \left( {y_{i} + \delta_{i}} \right)},{x_{j} - x_{i}}} \right\}} - \left( {\theta_{ij} + ɛ_{i}} \right)} \right\}^{2}}}\;} \\{{\bigtriangledown f} = {\overset{N}{\sum\limits_{i = 1}}\left( {{\frac{\partial f}{\partial x_{i}}{\hat{x}}_{i}} + {\frac{\partial f}{\partial y_{i}}{\hat{y}}_{i}} + {\frac{\partial f}{\partial ɛ_{i}}{\hat{ɛ}}_{i}}} \right)}}\end{matrix}$   where − Δϵ ≤ ϵ_(i) ≤ +Δϵ, for 1 ≤ i ≤ N

Further, in one embodiment, a higher similarity between the initialvalues and the ground truth locations may yield faster and more accuratereconstruction of the relative locations. To obtain initial values atless complexity, a linearly constrained quadratic polynomial programmingproblem may be formulated. An alternative cost function may beimplemented, which is the vertical distance between (x_(j), y_(j)) andthe line passing (x_(i), y_(i)) at a slope of tan θ_(ij). Consider FIG.12, which provides an exemplary geometry for the alternative costfunction.

Table 6 illustrates an accelerated cost function ƒ_(a). Of course, itshould be noted that the accelerated cost function ƒ_(a) shown in Table6 is set forth for illustrative purposes only, and thus should not beconstrued as limiting in any manner.

TABLE 6${f_{a}\left( {x_{1},y_{1},\ldots\;,x_{N},y_{N}} \right)} = {\sum\limits_{{({i,j})} \in O}\left\lbrack {\left\{ {{\left( {x_{j} - x_{i}} \right)\tan\mspace{11mu}\theta_{ij}} + y_{i}} \right\} - y_{j}} \right\rbrack^{2}}$

A quadratic programming algorithm may be used to find z minimizing ƒ_(a)subject to the same linear constraints hereinabove. θ_(ij) may be aconstant given from the observation.

Additionally, in one embodiment, for a large crowd implementation, CaaSmay be able to leverage the global positioning system (GPS) built in themobile devices. If a crowd's spatial dimension is greater than an errorrange of the GPS, the CaaS back end optimizer 530 may introduce extralinear inequality constraints about the devices' locations.Specifically, each device's (x,y) coordinate may be bounded by thedevice's locally measured GPS coordinate plus and minus a GPS errordistance.

Further, in one embodiment, when people sit on a seat or stand in astadium, their phones may be unable to see the screen of phones behind.One exemplary solution is to guide people to hold their phones tilted atthe angle of slope of the seats. To guide the right angle, the CaaSmobile application 502 may provide a tactile feedback when the phone istilted at a proper angle.

Further still, in one embodiment, the above methodology may beimplemented utilizing a divide and conquer strategy. For example, abreadth-first search from a device over a limited depth may isolate asubgroup of spatially adjacent devices. In this way, the entire group ofmobile devices may be divided into subgroups of adjacent devices, andthe localization may be completed in a linear time, or sub-linear byparallelism.

Also, in one embodiment, a solution to partial location changes of oneor more of a plurality of devices during implementation may be toinitiate spatially-isolated relative localization only for a subgroup ofdevices within the vicinity of a newly moved-in user. A move-out may bepaired with a move-in elsewhere, or may be identified upon closing theCaaS mobile app. To enable a newly moved-in device to detect nearbydevices, each CaaS mobile app may sporadically toggle its Bluetoothdiscoverability. In another embodiment, a device identifier may bedisseminated by watermarking it on the tiles of the device displays; anewly moved-in device may report a nearby seen identifier.

Additionally, each instance of the mobile application 502 may beresponsible for encoding its own identifier, performing localobservations, interacting with the cloud-side service 504, anddisplaying the timed sequence of tiles assigned on it. Second, thecloud-side service 504 includes two parts: (1) the front end coordinator524 which may be responsible for interacting with the mobile application502, collecting their local observations, generating and dispatching thetimed tile sequences to each mobile application, and (2) the back endoptimizer 530 which may construct a constrained optimization problembased on the local observations and may execute a solver instance basedon nonlinear programming techniques. As a result, individual users maybe accurately localized in a densely packed crowd, utilizing acollective localization technique based on mobile-side pair-wise visualobservations and cloud-side constrained optimization.

Further, to ensure accurate and scalable relative localizations, acollective localization technique is utilized based on mobile-sidepair-wise visual observations and cloud-side constrained optimization.For the mobile-side observations, a vision-based device identificationtechnique may encode a unique identifier into a color-transitionsequence on the device's display. All-user relative locations may thenbe reconstructed using distributed observations among mobile devices. Aconstrained optimization problem may then be applied to on theobservations set, which may complete localizations to achieve collectivevisualizations.

In this way, Card-stunt as a Service (CaaS), a service enabling aco-located crowd to instantly visualize symbols collectively over thecrowd using their mobile devices and cloud services, may be implemented.To address the relative localization challenges under the scenarios inindoor or outdoor public squares or stadiums where stationarylocalization infrastructure is unlikely, an architecture may be providedwith mobile vision sensing and cloud-side constrained nonlinearoptimization. The implementation may be robust and effective, and mayextend to consider topology changes of the crowds or providescalability. CaaS may create various new collective expressiveness forgroups of people.

Now referring to FIG. 6, a flowchart of a method 600 for implementing acard stunt as a service (CaaS) mobile application of a mobile device isshown according to one embodiment. The method 600 may be performed inaccordance with the present invention in any of the environmentsdepicted in FIGS. 1-5 and 7, among others, in various embodiments. Ofcourse, more or less operations than those specifically described inFIG. 6 may be included in method 600, as would be understood by one ofskill in the art upon reading the present descriptions.

Each of the steps of the method 600 may be performed by any suitablecomponent of the operating environment. For example, in variousembodiments, the method 600 may be partially or entirely performed byone or more servers, computers, or some other device having one or moreprocessors therein. The processor, e.g., processing circuit(s), chip(s),and/or module(s) implemented in hardware and/or software, and preferablyhaving at least one hardware component may be utilized in any device toperform one or more steps of the method 600. Illustrative processorsinclude, but are not limited to, a central processing unit (CPU), anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA), etc., combinations thereof, or any other suitablecomputing device known in the art.

As shown in FIG. 6, method 600 may initiate with operation 602, where acard stunt as a service (CaaS) mobile application of a mobile deviceidentifies vision code data from additional devices, utilizing ahardware camera of the mobile device. In one embodiment, the Caas mobileapplication may identify the vision code data from the other deviceswhile outputting vision code data itself utilizing a display of themobile device.

Additionally, method 600 may proceed with operation 604, where the CaaSmobile application of the mobile device identifies an orientation of themobile device, utilizing one or more sensors of the mobile device.

Further, method 600 may proceed with operation 606, where the CaaSmobile application of the mobile device calculates local observations,including relative angles and orientation, and sends the localobservations to a cloud-side CaaS service.

Further still, method 600 may proceed with operation 608, where the CaaSmobile application of the mobile device receives timed tile sequencesfrom the cloud-side CaaS service, in response to the sending of thelocal observations.

Also, method 600 may proceed with operation 610, where the CaaS mobileapplication of the mobile device outputs the received timed tilesequences, utilizing a display of the mobile device.

In this way, the CaaS mobile application may perform visual observationsof the environment surrounding the mobile device, submit suchobservations to the cloud-side CaaS service, and display timed tiles onthe mobile device in order to effect a card stunt implementation inconjunction with a plurality of other mobile devices.

Now referring to FIG. 7, a flowchart of a method 700 for implementing acard stunt as a cloud-side (CaaS) service is shown according to oneembodiment. The method 700 may be performed in accordance with thepresent invention in any of the environments depicted in FIGS. 1-6,among others, in various embodiments. Of course, more or less operationsthan those specifically described in FIG. 6 may be included in method600, as would be understood by one of skill in the art upon reading thepresent descriptions.

Each of the steps of the method 700 may be performed by any suitablecomponent of the operating environment. For example, in variousembodiments, the method 700 may be partially or entirely performed byone or more servers, computers, or some other device having one or moreprocessors therein. The processor, e.g., processing circuit(s), chip(s),and/or module(s) implemented in hardware and/or software, and preferablyhaving at least one hardware component may be utilized in any device toperform one or more steps of the method 700. Illustrative processorsinclude, but are not limited to, a central processing unit (CPU), anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA), etc., combinations thereof, or any other suitablecomputing device known in the art.

As shown in FIG. 7, method 700 may initiate with operation 702, where acloud-side card stunt as a service (CaaS) service receives localobservations from a plurality of CaaS mobile applications associatedwith a plurality of mobile devices. In one embodiment, the localobservations may be received at a front end coordinator portion of thecloud-side CaaS service.

Additionally, method 700 may proceed with operation 704, where thecloud-side CaaS service determines relative locations for the pluralityof mobile devices by performing constrained optimization, utilizing thelocal observations. In one embodiment, the constrained optimization maybe performed at a back end optimizer of the cloud-side CaaS service.

Further, method 700 may proceed with operation 706, where the cloud-sideCaaS service determines timed tile sequences for each of the pluralityof mobile devices, utilizing the relative locations for the plurality ofmobile devices. In one embodiment, the back end optimizer may providethe front end coordinator with the relative locations, and the front endcoordinator may then determine the timed tile sequences.

Further still, method 700 may proceed with operation 708, where thecloud-side CaaS service dispatches the timed tile sequences to each ofthe plurality of mobile devices.

In this way, the cloud-side CaaS service may utilize observation datafrom each of the plurality of mobile devices to determine a location ofeach of the plurality of mobile devices relative to each other, and sendtimed tiles to each of the plurality of mobile devices based on theirlocations in order to effect a card stunt implementation.

Now referring to FIG. 8, an exemplary card stunt as a service (CaaS)visualization 800 is shown according to one embodiment. As shown, theCaaS visualization 800 is implemented utilizing a first plurality ofdisplays 802A-J of mobile devices in addition to a second plurality ofdisplays 804A-Y of mobile devices. In one embodiment, each of the firstplurality of displays 802A-J and the second plurality of displays 804A-Yincludes a display portion of a mobile device (e.g., a screen of thedevice, etc.). For example, each of the first plurality of displays802A-J and the second plurality of displays 804A-Y may be held above ahead of an individual user (e.g., a user within a group of users).

Additionally, in one embodiment, each of the first plurality of displays802A-J display a first color (e.g., a first colored tile), and each ofthe second plurality of displays 804A-Y display a second color (e.g., asecond colored tile) different from the first color. In anotherembodiment, for each of the first plurality of displays 802A-J and thesecond plurality of displays 804A-Y, the color presented by the displaymay be indicated as part of a timed tile sequence received by thecorresponding mobile device, where the timed tile sequence received byeach mobile device is based on a determined relative location of themobile device within the plurality of mobile devices.

In this way, an observer separate from the group of users that observesthe first plurality of displays 802A-J and the second plurality ofdisplays 804A-Y from a distance may view a composite image (in thisexample, the word “HI”) resulting from the different colored tiles

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein includes anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which includes one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

Moreover, a system according to various embodiments may include aprocessor and logic integrated with and/or executable by the processor,the logic being configured to perform one or more of the process stepsrecited herein. By integrated with, what is meant is that the processorhas logic embedded therewith as hardware logic, such as an applicationspecific integrated circuit (ASIC), a FPGA, etc. By executable by theprocessor, what is meant is that the logic is hardware logic; softwarelogic such as firmware, part of an operating system, part of anapplication program; etc., or some combination of hardware and softwarelogic that is accessible by the processor and configured to cause theprocessor to perform some functionality upon execution by the processor.Software logic may be stored on local and/or remote memory of any memorytype, as known in the art. Any processor known in the art may be used,such as a software processor module and/or a hardware processor such asan ASIC, a FPGA, a central processing unit (CPU), an integrated circuit(IC), a graphics processing unit (GPU), etc.

It will be clear that the various features of the foregoing systemsand/or methodologies may be combined in any way, creating a plurality ofcombinations from the descriptions presented above.

It will be further appreciated that embodiments of the present inventionmay be provided in the form of a service deployed on behalf of acustomer to offer service on demand.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. Thus, the breadth and scope of a preferred embodiment shouldnot be limited by any of the above-described exemplary embodiments, butshould be defined only in accordance with the following claims and theirequivalents.

What is claimed is:
 1. A computer-implemented method, comprising:identifying a plurality of mobile devices; sending a unique identifierto each of the plurality of mobile devices; receiving results of mobilevision sensing from each of the plurality of mobile devices, where theresults of mobile vision sensing include, for each of the plurality ofmobile devices, an identification of the unique identifier of othermobile devices of the plurality of mobile devices that are within afield of view of the mobile device; determining a relative location ofeach of the plurality of mobile devices, utilizing the results of mobilevision sensing; resampling textual output, graphical output, or textualand graphical output to fit a number of the plurality of mobile devicesto create visualization data; and sending the visualization data to eachof the plurality of mobile devices, based on the relative location ofeach of the plurality of mobile devices.
 2. The computer-implementedmethod of claim 1, wherein each of the plurality of mobile devices is ina possession of a unique individual located within a single crowd ofpeople.
 3. The computer-implemented method of claim 1, whereindetermining the relative location of each of the plurality of mobiledevices includes performing relative localization on each of theplurality of mobile devices in order to determine a co-located group ofthe mobile devices.
 4. The computer-implemented method of claim 1,wherein the results of mobile vision sensing further include a pluralityof tuples, where each of the plurality of tuples includes the uniqueidentifier of a first mobile device of the plurality of mobile devices,the unique identifier of a second mobile device that is within a fieldof view of the first mobile device, and a horizontal angle defined by: afirst line perpendicular to an image plane of a camera of the firstmobile device, and a horizontal projection of a second line connecting acenter of the camera of the first mobile device and a center of a screenof the second mobile device.
 5. The computer-implemented method of claim1, further comprising assigning individual cues to each of the pluralityof mobile devices, based on the relative location of each of theplurality of mobile devices, where the individual cues instruct a userof each of the plurality of mobile devices to manually display one ormore cards.
 6. The computer-implemented method of claim 1, wherein thevisualization data includes a single tile or a timed sequence of tilesthat are sent to each of the plurality of mobile devices.
 7. Thecomputer-implemented method of claim 1, wherein constrained nonlinearoptimization is performed within a cloud computing environment,utilizing the results of mobile vision sensing, in order to determinethe relative location of each of the plurality of mobile devices.
 8. Thecomputer-implemented method of claim 7, wherein determining the relativelocation of each of the plurality of mobile devices further comprises:accelerating the constrained nonlinear optimization by formulating alinearly constrained quadratic polynomial programming problem, solvingthe linearly constrained quadratic polynomial programming problemutilizing a quadratic programming algorithm to obtain an approximatesolution, and sending the approximate solution as initial values to thelinearly constrained quadratic polynomial programming problem.
 9. Thecomputer-implemented method of claim 1, further comprising: pixelatingtextual output, graphical output, or textual and graphical output into amatrix of tiles; and assigning each tile to one of the plurality ofmobile devices at a location matching a respective location within thematrix of tiles.
 10. A computer program product for providingvisualization data to a plurality of mobile devices, the computerprogram product comprising a computer readable storage medium havingprogram instructions embodied therewith, wherein the computer readablestorage medium is not a transitory signal per se, the programinstructions executable by a processor to cause the processor to performa method comprising: identifying the plurality of mobile devices,utilizing the processor; sending a unique identifier to each of theplurality of mobile devices, utilizing the processor; receiving,utilizing the processor, results of mobile vision sensing from each ofthe plurality of mobile devices, where the results of mobile visionsensing include, for each of the plurality of mobile devices, anidentification of the unique identifier of other mobile devices of theplurality of mobile devices that are within a field of view of themobile device; determining, utilizing the processor, a relative locationof each of the plurality of mobile devices, utilizing the results ofmobile vision sensing; resampling, utilizing the processor, textualoutput, graphical output, or textual and graphical output to fit anumber of the plurality of mobile devices to create visualization data;and sending, utilizing the processor, the visualization data to each ofthe plurality of mobile devices, based on the relative location of eachof the plurality of mobile devices.
 11. The computer program product ofclaim 10, wherein each of the plurality of mobile devices is in apossession of a unique individual located within a single crowd ofpeople.
 12. The computer program product of claim 10, whereindetermining the relative location of each of the plurality of mobiledevices includes performing, utilizing the processor, relativelocalization on each of the plurality of mobile devices in order todetermine a co-located group of the mobile devices.
 13. The computerprogram product of claim 10, wherein the results of mobile visionsensing further include a plurality of tuples, where each of theplurality of tuples includes the unique identifier of a first mobiledevice of the plurality of mobile devices, the unique identifier of asecond mobile device that is within a field of view of the first mobiledevice, and a horizontal angle defined by: a first line perpendicular toan image plane of a camera of the first mobile device, and a horizontalprojection of a second line connecting a center of the camera of thefirst mobile device and a center of a screen of the second mobiledevice.
 14. The computer program product of claim 10, further comprisingassigning, utilizing the processor, individual cues to each of theplurality of mobile devices, based on the relative location of each ofthe plurality of mobile devices, where the individual cues instruct auser of each of the plurality of mobile devices to manually display oneor more cards.
 15. The computer program product of claim 10, wherein thevisualization data includes a timed sequence of tiles that are sent toeach of the plurality of mobile devices.
 16. The computer programproduct of claim 10, wherein constrained nonlinear optimization isperformed within a cloud computing environment, utilizing the results ofmobile vision sensing, in order to determine the relative location ofeach of the plurality of mobile devices.
 17. The computer programproduct of claim 16, wherein determining the relative location of eachof the plurality of mobile devices further comprises: accelerating,utilizing the processor, the constrained nonlinear optimization byformulating a linearly constrained quadratic polynomial programmingproblem, solving, utilizing the processor, the linearly constrainedquadratic polynomial programming problem utilizing a quadraticprogramming algorithm to obtain an approximate solution, and sending,utilizing the processor, the approximate solution as initial values tothe linearly constrained quadratic polynomial programming problem. 18.The computer program product of claim 10, wherein the visualization datarepresents textual output, graphical output, or textual and graphicaloutput that is pixelated into a matrix of tiles.
 19. A system,comprising: a processor; and logic integrated with the processor,executable by the processor, or integrated with and executable by theprocessor, the logic being configured to: identify a plurality of mobiledevices; send a unique identifier to each of the plurality of mobiledevices; receive results of mobile vision sensing from each of theplurality of mobile devices, where the results of mobile vision sensinginclude, for each of the plurality of mobile devices, an identificationof the unique identifier of other mobile devices of the plurality ofmobile devices that are within a field of view of the mobile device;determine a relative location of each of the plurality of mobiledevices, utilizing the results of mobile vision sensing; resampletextual output, graphical output, or textual and graphical output to fita number of the plurality of mobile devices to create visualizationdata; and send visualization data to each of the plurality of mobiledevices, based on the relative location of each of the plurality ofmobile devices.